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ABSlfiACT 

A lajor problei inherent in the synthesis of the 
Hpdel Training Project (HTP) evaluation reports cited herein is tlie 
aijsence of the following: (1) a forial project evaluation design, (2) 
a set of adegnate specifications or standards upon which to assess 
project activities, (3) concensus asong project staff on specific 
criteria for project evaluation. Conseguently, lost evaluation 
efforts were ad hoc, lacking in overall continuity, and based on very 
general standards for evaluation* These problems were recognized 
internally by the staff, and were noted in a nuiber of HTP documents. 
The continuity that did exist was of two types: first, the external 
review teais had repeat leibers; second, the external review team 
findings were fcllowed up to attempt to document project 
responsiveness; and third, the guarterly and yearly reports to QSOE 
used a reporting format that sniiKd the project activities 
chronologically. Thus, although there was some continuity, the lack 
of an overall evaluation design made the task of compiling a cohesive 
synthesis impossible. Therefore, this document reports ikumerous 
evaluation efforts that are little related to each other, which 
reflects the conditions of ad hoc evaluation that characterized the 
HIP. (Author/HV) 
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INTRODUCTION 

A major problem Inherent in the synthesis of the HTP evaluation 
reports cited herein !s the absence of the following: 
K A formal project evaluation design 
2* A set of adequate specifications or standards upon 

which to assess project activities* 
3* Concensus among project staff on specific criteria 
for project evaluation* 
Consequently, most evaluation efforts wefe ad hoc , lacking in overall 
continuity, and based on very general standards for evaluation* These 
problems were recognized Internally by the staff, and were noted 
in a number of HTP docwnents such as the In-House Quarterly Report 
(Eye, et al*, 197^* C])* The continuity that did exist was of two 
types: first, the external review teams had repeat members; second, 
the external review team findings were followed up to attempt to 
document project responsiveness; and third, the quarterly and yearly 
reports to USOE used a reporting format that summed the project 
activities chronologically* Thus, although there was seme continuity, 
the lack of an overall evaluation design made the task of comprllng 
a cohesive synthesis impossible* Therefore, this document reports 
numerous evaluation efforts that are little related to each other, 
which reflects the conditions of ad hoc evaluation that characterized 
the MTP* 



CUSSIFI CATION PROCESS OF EVALUATION OOCUMENTS 



This report has as its organizational base a physical classification 
of documents which is as follows: 

A. htlP Reports are any evaluation report produced by the project 
for an external agency or individual, other than the consortium* 

B. Proposal s are a contractor's request for continued funding. 

C. Internal Evaluation Reports are any reports produced by the 
project for any Evaluation Center staff mefrtber. 

0. Site Visits and External Reports are evaluation docwnents 
produced by any external source* 

E. Consortium Reports are any reports produced by the project for 
any consortium staff member. 

The documents clitssified by the five groups wer the.i analyzed for 

evaluation material pertinent to the KTP*s initial proposed objectives, 

and the Standards under each Objective* The reporting format uses the left 

column for a brief content statement of material, The right column gives 

the bibliographic citation. For complete information, the reader needs 

the fol lowing I ist: 

List of Oocuments 

A, KTP freports 

A - Program to Operational ize a New Training Pattern for Training 
Evaluation and Innovation Process Personn<5l in Education, 
Technical Progress Report, October II, I97K 

Aj ' Project to design New Pattern for Training Research, Development, 
Innovation Process, and Evaluation Personnel in Education, 
Semi Final Report, November 13, 1970. 

A, - Proposal to Design New Patterns forTralning Research, Development, 
Demonstration/Dissemination and Evaluation Personnel in Education, 
Context Evaluation Report, April I3j I97K 

Ai^ - Proposal to Design New Patterns for Training Research, Develop- 
ment, Demonstration/Dissemination and Evaluation Personnel in 
Education, Progress Report #2, Context Evaluation, September 10, 

1970. 5 
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Aq ' Program to Operational ize a New Training Pattern for Training 
Evaluation Personnel tn Education, Technical Progress Report, 
January 31, 1972. 

Ag - Program to Operational l2e a New Training Pattern for Training 
Evaluation and Innovation Process Personnel to Education, 
Technical Progress Report, June 18, 1971. 

A- - Program to Operational 1 2e a New Training Pattern for Training 
Evaluation and Innovation Process Personnel in Education, 
Technical Progress Report, March 18, 197K 

B. Proposal s 

Bj * Proposal to Oesign New Patterns for Training Research, Oevetopment 
Oemonstration/Oissemination, and Evaluation Personnel in Education, 
May 22, 1970. 

^2 ' Proposal to Oesign New Patterns for Training Research, Oevelop* 
ment, Oemonstration/Oissemination and Evaluation Personnel in 
Education, Final Report and Proposal, Oecember 18, 1970. 

C. Internal Evaltiatlon Reports 

Cj - In-House MTP Quarterly Report, March 29, 1972. 

C2 * Oecisions Based upon In-House MTP Quarterly Repcrt: Student 
Survey (no date) . 

Cj - PROB {no date) . 

- Barger Model/Placement Sub*System (no date). 

Cy ' Barger Model/Sub-System Processes (no date). 

Cg - Barger Model/Learning Experiences Sub-System {no date). 

Cq - MTP Position Paper on Student Selection {R. R. Barger), 

^ May 26, 1972. 

CjQ - Problems, Observations, Solutions? {0. Reinhard), April 6, 1972. 

Cj j - Profiles of the Programs of Evaluation Majors, February 1, 1972. 

' Profile of New Students, November 23, 1971, 

Cjg - In-House MTP Quarterly Report: Student Survey, March 28, 1972. 

CjQ - Student Attitudes Toward Educational Evaluation, MTP Internal 
Evaluation Unit, August M, 1972. 
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Internships - Student Attitude {no date). 



^28 



KTP Scope of Work Process Evaluation Report, February 1, 1972, 

Proportuion of Credit Hourse In Four Technical Areas to Total 
Number of Credit Hours Accumulated Since Autumn I97I Through 
the Present Quarter (no date). 



D. Site Visits and External Reports 



^I 

h 

Or 



OE Site Visit October 26^27, 1971, Exit Interview. 

MTP Recommendations from HEW Team, Michael Scriven, Frank Chase, 
Eva Baker (no date). 

Egon G. Guba, Consultant's Report (covering visit of May 25*27, 
I97I), June II, 197L 

^f^P Responses To Outside Evaluation, February, 1972. 
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Letter from John C, Egermeier and Susan S. Kletn to Daniel 
L. Stufflebeam, November 17, I97I. 

Michael Scrlven, MTP Outside Review Panel, visit of July 29* 
30, I97I, Report of September 7, 197K 

Eva Baker, KTP Outside Review Panel, visit of Muly 29*30, I97K 

Francis S. Chase, Report on the Site Visit to the MTP in Evalua- 
tion of the Evaluation Center of OSU, I97K 

John B. Peper, Chairman's Report of the Site Visit to OSU 
Model Training Project, October 13-1^, 1972. 



Consortium Reports 
E2 - MTP Consortium Meeting, Columbus, Ohio^ July 10-11, 1972. 



Et 
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The Council of the Great City Schools Site Visit Report 
(Jack P. Taylor and others - no date). 

Saginaw Site Visit Report (Marvin Oswson and others - no date). 

The Cincinnati Public School System Site Visit Report 
(Marvin Gervirtz and others * no date). 

Institute for Educational Development Site Visit Report 
(Jerry R. Baker and others - no date). 

MTP Consortium Meeting, December 1-3, 1971: Process Evaluation 
Report, December II, 1971- 

Evaluation of Shanty Creek Meeting, July 20, 1971. 
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General Obtecttve I t TO CONCEPTUALIZE A GENERAL MODEL 
FOR TRAINING EDUCATIONAL EVALUATORS. 



5 



(A) SPECIFY THE FOLLOWING PROCEDURES; 
RECRUITMENT, APPLICATION, SELECTION 
ADVISORY COMMITTEE, PROGRAM DESIGNING, 
PROGRAM EXPERIENCE AND MONITORING, 
PROGRAM COMPLETION, PLACEMENT, AND 
RE-ENTRY. 

Figure I contains the 9enera1 oiodel which 
will be the basis for the proposed training 
program, ♦ includes five major stages in the 
training process, , ^supported by five program 
components , 

, *no national symposium wi U be held 
to assess the Bargar model* ♦ 

Standards of evaluation are presented. 



Context information ts presanted as a 
basis for this standard. 

Baseline information for developing 
a training program is presented, 

(B) PROVIDE WELL DEFINED BASES FOR 

DEVELOPMENT OF PROCEDURES, DOCUMENTS, 
ANO ACTIVITIES NECtSSARY FOR nS 
IMPLEMENTATION* 

Program development section will consist 
of following: I) work breakdown of activities 
and 2) time sequency description. 

The model does not specify materials, 
however, it sets ctit areas. * ,few people 

8 



Qbiecttves 
Input Evaluation 
Structuring Decisions 
November 13, 1970 
p. 227 ff. A2 



In-House ttuarterly Report 

Marcfr2ir7T972 

p. I 

^18 

Technical Progress Reports 
Volume 1 

October 1!, 1971 
p. 98 

Context Evaluation Report 
April 13, 1971 
Section II, Part II 
p. 63 ff. A3 

Progress Report #2 
September 10, 1970 
p. 1 ff. 



Semt-Ftnal Report 
November 13, 1970 
p.2kS ff. A2 



In-House _&uarterlv Report 
March 2k, 1972 
p. I 
Cl8 
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other than Bargar understand the intricacies 
of the node) . 

Flow charts of training system are 
presented. 

The model is not convnunicable in it's 
present form. 

Standards for evaluation are presented^ 



(C) WILL BE SUFFICIENTLY GENERALIZABLE 
SO THAT IT CAN BE INSTALLED AND 
OPERATED BY GROUPS OF FIELD AND 
UNIVERSITY AGENCIES BEYOND O^iE 
CONSORTIUM OPERATING THE PRESENT 
PROJECT. 

One test for general izabi 1 ! ty was to be 
the national symposium that will not be held. 

Standards for evaluation are presented* 



(0) WILL BE GENERALIZABLE FOR TRAINING 

OF PERSONNEL IN THE AREAS OF RESEARCH, 
DEVELOPMENT, AND INNOVATION PROCESSES, 
IN ADDITION TO EVALUATION. 

The training component pictured in Figure 

k shows blocks for four types of training: 

evaluation, innovative processes (diffusion), 

research , and development. 



Bargar Model papers 
Learning Experiences Sub-System 
C7, C8 

HTP Consortium Meeting 
Process Evaluation Report 
December I97t 
p. 17 

Technical Progress Report 
Volume I of I I Volumes 
October 11, 1971 
p. 98 



ln~House Quarterly Report 

March 24, 1972 

P*l 

Technical Progress Report 
Volume t of t 1 Volumes 
October 11, 1971 
p. 99 
Al 



Final Report 

Volume 1 of I t Volumes 

p. 17 
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There is no evidence that the mode) has 
been adopted by any outstde system or agency* 
Revised model has yet to be subjected to an 
outside review. 

Standards for evaluation are presented. 



(E) WILL BE INTERNALLY COHSfSTEHT. 
Assessment of this standard is not 
being done* 

Standards of evaluation are presented, 



(F) INTERPRETABLE TO GENERAL EDUCATION 
AUDIENCES. 

There is a lack of understanding by 
educational specialists at the December, 1971 
Consortium meeting* This strongly suggests 
less sophisticated audiences would not fare 
any betf^r (in understanding)* 

Standards of evaluation are presented. 



(G) SUGGEST CRITERIA FOR OPERATIONAL 
TESTING, 

Criteria have not been stated for this 
standard* 

10 



In'House Quarterly Report 
March 2ii, 1971 
p. 2 



Technical Progress Report 
Volune 1 of M Volunes 
October II, 1971 

p. too 

Al 



In'House ttuarterly Report 
March Zk, 1972 

Cl8 

Technical Progress Report 
Volune 1 of I t Volumes 
October 11, 1971 
p. 101 

^^1 



In-House quarterly Report 
March 21*, I97l 
p. 2 



Technical Progress Report 
Volume ! 

October tl, 1971 
p. 101 



In-House quarterly Report 
March ?A, l3Ti 
p. 2 
Cl8 
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Standards of evaluation are presented. 



Technical Progress Report 
Volume I 

October 11, 1971 
p. 102 



(H) WILL BE VIABLE BOTH IN TERMS OF FUNDING 
AND THE USUAL ACADEHtC CONSTRAINTS TO 
8E FOUND IN GROUPS DF F1£LD AND UN tVQ^- 
SITY AGENCIES OTHER THAN THOSE PRE- 
SENTLY INVOLVED IN THE HODEL TRAINING 
PROJECT. 

Alternative Strategies for the Support 
of Students and Parent of Instructional Fees 
are presented. 

Data reported by agencies Relating to 
Training is shown. 

There is no data to relate to this 
standard. 

Standards of evaluation are presented. 



Semi-Ftnal Report 
November 13, 1970 
p. 308 ff. 



Context Evaluation Report 
April (3, 1971 
pM ff . A3 

In-House Quarterly Report 
March 24, 1972 ~ 
p. 2 
Ci8 

Technica^ Progress Repor t 
Volume t 

October 11, 1971 
p. 102 
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General Qbtectlve 2 : TO OPERATIOHALlZE A 
GENERAL MODEL FOR TftAtNING EOUCATtONAL 
EVALtMTORS, 

(A) THE MATERIALS AND SYSTEMS THtS WILL 
REQUIRE WtLL INVOLVE REALtSTtC AND 
EFFECTIVE RECRUITMENT PROCEDURES 
AND MATERIALS TO 8E DEVELOPED* 

During the October, 1971 USOE exit 
interview, John Hopkins Indicated that re- 
cruitment procedures were '^ilsslng a poten- 
tial poot of highly talented prospects** for 
the MTP* The present composition of MTP 
students indicated that recruitment procedures 
weFe probably aimed at drawing students who 
were already involved In educational evaluation. 

Recruitment materials (brochures, 

transparencies, and ^ slide and tape presen*" 

< 

tation) were revised, based on student, 
faculty, and consortium personnel feedbacl<, 
and prepared ^or distribution. 

During 1972, recruitment procedures 
were informal and heavily relied on the 
subjective judgement and experience of 
adjunct professors* 

(B) the materials and systems THtS WILL 
REQUIRE WILL INVOLVE SELECTION PROCEDURES 
AND MATERIALS* 

John Egermeier touched upon the student 
selection problem probably to be encountered 
during the USOE October, 1971 visit. He 

12 



A 

P,2 



A 

p. 37 



p. 22 



h 

8 

p.2 



tndtcated that educators have a long tradition 
of tratning only other educators to work ^1 
education and that the KTP could probably 
**pick up a lot of strength by increasing 
the diversity of our group*" 

While special materials and systems 

B 

were developed to aid selection, these were p*22 
never well Integrated with University materials* 

The positron paper on student selection Cg 

B 

indicated student attributes that were considered P*2,3 

important to educational evaluators. These 

attributes were informally identified through 

several sources, Including general readings, 

discussions with the colleagues tn evaluation, 

and through the author's personal perceptions 

of those factors that appeared to be most 

important. The attributes were subsequently 

classified into four categories: ]) cognitive 

characteristics, 2) affective characteristics, 

3) professional-orientation characteristics, 

and ^) work character istics > 



As 



The student survey collected some Student Survey 

B 

unanticipated data regarding the selection P*I5 
of students into the MTP as opposed to the 
Library Project* During the course of the 
interview, it was discovered that MTP students 



on the Library Project felt separated from 
the MTP students working under Darrell Root's 
jurisdiction. The perceptions of Library 
Project students were unanimous. It was also 
noted that all students on the Library Project 
were members of ethnic minorities. The 
evaluators concluded that while the ratio of 
ethnic distribution was statistically possible, 
it was socially intolerable* 

(C) THE MATERIALS AND SYSTEMS THIS WILL 
REQUIRE WILL INVOLVE INSTRUMENTS FOR 
TRAINEE DIAGNOSIS. 

Diagnosis of student learning needs 

have largely been conducted individually 

by the traditional advisement procedures. 

Other than SAES, developed largely 
over the spring and summer of 1972, trainee 
diagnosis was primarily an Informal process 
attended to by the trainee's advisor* Eva I * 
uation of SAES does not appear in any MTP 
documents prepared prior to the phase-out 
per iod . 

(D) THE MATERIALS AND SYSTEMS THIS WILL 
REQUIRE WILL INVOLVE SPECIFIC 
CURRICULUM MODULES PERTAINING TO 
CONTENT OF THE TRAINING MODEL THAT 
MIGHT BE TRANSMITTED TO STUDENTS 
VIA INSTRUCTIONAL PACKAGES, FIELD 
TESTING, OR SHORT-TERM INSTITUTE- 
TYPE TRAINING. 



C2 
C 

p. 38 

'^Diagnosis of student learning 
needs have been conducted 
individually by students' major 
advisors." 
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A Context and Input evaluation found 
that there was a lack of read! I y ava t lable, 
sYstematicaUY developed, validated materials 
for training educational researchers of any 
type. 



Prototype versions of the three slide/ 
tape presentatlr^ns were viewed by adjunct 
professors at the Dallas consortium meeting. 
This enabled the developers to obtain Formative 
feedback for refinement purposes. Input was 
also acquired by soliciting critiques from 
audio Visual specialists, content experts, 
and consortium staff members. These data 
were additionally used for further developmental 
modifications. 



B2 
D 

"the systematic study and planning 
that resulted in the proposed 
HTP clearly identified require- 
ments for successful development ^ 
and operation of the program. 
As the study and planning evolved, 
It became obvious that field 
agencies consistently lacked 
sets of materials for training 
staff for evaluation, development, 
and innovative processes." 

A6 
D 

P*9 



During a July, 1971 site visit, Michael 
Scriven pointed out that planning materials 
would improve greatly if they simulated the 
needs of new students* Scriven also under" 
scored the urgent need to write behavioral 
specifications and then tests and content 
for a couple of core modules explicating 
evaluation ski I Is. 



V 
p.if 

'The planning materials under 
both headings are substantial 
and useful or promising, but 
i think they too would be im- 
proved more quickly as a result, 
not of further cycles of staff 
discussions of them as planning 
documents, but as a result of 
simulating the needs of the 
entering students, role-playing 
entering students, and asking 
what it is that they are going 
to need on Day I when they enter 
the door for the first time.*' 
(Scriven, D7, D* p. 2) 
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Scriven also noticed a dtscrepancy between ^7 

0 

an MTP goal and the adopted teaching strategies. p*8 
He discovered that MTP training materials 
were using a deductive approach to training 
by tnttiatly sequencing baste concepts and 
theories prior to putting the students to work 
when materials indicated that students would 
be exposed to an inductive approach. Scriven's 
final commentary on instructional materials 
focused on the urgency for devel opl ng modules 
whtch would eliminate much of the redundancy 
and irrelevance in ? course**work approach. 

Eva Baker, a second member of the O7 
advisory site team, also addressed some of p. ID 

her comments to the nature and developmental 
status of the materials development component 
during the July, I97l advisory panel site 
visit. She indicated that the planned development 
procedures were much too complex for available 
resources* She recommended that reference 
to a simpler developmental model, with fewer 
decision points, would be more consonant with 
project resources* 

A panel discussion among MTP staff 

0 

and the advisory site visit members lead to p^^ 

'*Such a requirement is both 
an examination of criteria for MTP materials i 1 1'thought*out and unjusttf 

16 



selection. The discussion revealed that the 
first criterion was compatible with the CI?? 
model. Scriven's report expressed concern 
with this criterion which could potentially 
limit diversity of student exposure to other 
evaluation models. 

During another site visit, Egon Guba 
noted that the materials development component 
failed to give a high priority to tailoring 
materials to the particular audience for which 
they were intended. His assessment was made 
subsequent to viewing two slide/tape presen* 
tations which he felt failed to address their 
primary audience. 

While Guba forecasted serious problems 
for the instructional development component, 
these were primarily attributable to highly 
inadequate budgetary allocations and staffing 
problems . 

Guba presented a rationale for adopting 
flexi ble modular train ing mater iais . Ue 
recormended inclusion of explicit performance 
objectives and that '*the levels of the program 
be linked in some kind of explicit career 
lattice structure." Guba's report showed 
concern for the absence of these attributes 



Accepting the Center's verbal 
assurance that the actual 
situation encourages much more 
diversity than this suggests, 
its worth explicated the 
difficulty. Some training 
programs must absolutely not 
be indoctrination programs 
and this one is an example*** 
(07, 0, p.^). 



0 

p. 17 

"the two examples of slide/ 
tapes which I saw mtssed this 
mark materially. ?art of the 
problem ts, I bel leve, traceable 
to the low Dudget." (Dif, 0, p. 17) 



0 

p J6 

*'l have already registered 
several doubts about the 
potential of this component 
for productive operation, due 
primarily to the low budget 
allocation and the staffing 
problem (Oi^, 0, p. 16) 

i" 

p JO 

'Where Is the explicit commitment 
that the materials to be developed 
should be modular in form? Where 
is the listing of performance 
objectives (or where is the 
process spelled out whereby the 
performance objectives will be 
derived)? Where is the career 
lattice spelled out? How is the 
individualization to occur? 
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underscoring that these considerations should These steps seem to me to be 

necessary to delineate in 

have been delineated in advance of production* advance as much as possible^ 

to avotd the production of 
training materials that are 
randomly generated and that 
will not fit together tn any 
articulated way*" (0^, 0« P*IO) 

Identification of curriculum packages Ay 

0 

pertaining to content of the training model p*5 
was made based on four principle points 
involving: I) a determtnatton that the 
information contained in the packages will 
need to be presented at the first general 
consortium meeting, 2) a need for consortium 
representatives to present similar information 
to individuals within their own agencies, 
3) a determination that a knowledge of the 
content of these modules will be needed 
by all trainees, and k) the des i reabi t i ty 
of these modules for explicating the training 
program to a wide variety of interested people* 
A si ide/tape format was selected for the 
packages in order to facilitate use by an 
individual or by an entire group. 



The materials development component 
undertook the development of three seminar 
packages based on the conclusions of a 
consortium task force. These were: 1) a 
modulo on Evaluation of Institutionalization 

18 



p. 23 

^'Responding to the expressed 
needs of a consortium task force 
cofwnissloned to develop initial 
seminars for all MTP project 
trainees. Jack Sanders scheduled 
three "seminar packages" for 
production." {A5, 0, p. 23) 



and Change Strategies, 2) a survey of the 
Theories, Functions and Methods of Evaluation, 
and 3) seven interrelated modules for an 
introductory measurement course. 

During a later USOE site visit, John 

Hopkins, synthesizing the teams comments 

regarding the materials development operation, 

noted that tn view of the relative lack of 

resources tt would indeed be wise to put less 

emphasis on materials development* It was 

then suggested that available talent concentrate 

on the production of a few high quality modules 

deemed of central importance to the operation 

of this particular training program. 

(E) THE MATERIALS AND SYSTEMS THIS WILL 
REQUIRE WILL INVOLVE MATERIAL FOR 
PLANNING STUDENTS* PROGRAMS* 

The absence of an accepted process- 
product evaluation design resulted in fragmented 
ad hoc evaluations of the materials and systems 
developed to facilitate student program planning. 
Attempts to interrelate these isolated assessments 
would produce an artificial and deceptive 
picture of the evaluation endeavors* 

One of the earliest attempts to help 
students plan their programs involved feedback 
from consortium adjunct professors. They 
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were requested to reflect upon their perceptions 
of the core evaluation staff t>eeds of their 
agencies* Thier feedback revealed that there was 
almost an equal number of positions calling for 
skills in the problem delineation and providing 
role, as there were in roles requiring the more 
traditional types of technical obtaining skills,^ 

Profiles of the Programs of Evaluation Cn 

E 

majors was a document composed of raw data 
detailing individual program profiles of Evat- 
uation majors, tt was compiled with the express 
purpose of facilitating program planning 
for MTP students* The document ts devoid 
of a descriptive text. 

During an indepth student survey, C)8 

E 

students surveyed repeatedly pointed out that p.^ 

"You find out what you think 
Center staff do not have an adequate cognizance you want to take and get you 

advisor to sign it/' (C27» E 

of university offerings or their appropriateness* 
This was undoubtedly attributable to the fact 
that staff advisors were new to Ohio State, 
About ftfty percent of the respondents felt 
there was little to no formal advisement 
procedure. 



A later analysis of coursework in the C28 
technical areas of evaluation among MTP students 
programs was undertaken to determine whether 

20 
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student"^, were engaged in those types of courses 
on campus* The analysis divWed technical 
coursework into the general areas of: 
1) statistics and design, 2) evaiuation, 
3) measurement, and ^) research. The data 
revealed that MTP students coursework invoived 
an average of seventvtwo percent across the 
four areas. The largest proportion of course- 
work was in statistical and design courses. 
The smallest portion of coursework occurred 
In research. The accuracy of subcategory 
course classification was somewhat ambiguous, 
however, the overall analysis did confirm 
that MTP str'^'^nts were taking a good number 
of courses In the more technically oriented 
areas. 

The PRDB operation attempted to help 
students plan their programs by shepardlng 
the evaluation of O.S.U. courses by MTP 
students In order to provide students with a 
more indepth look at courses relevant to their 
career goa 1$ . 

(F) THE MATERIALS AND SYSTEMS THIS WILL 
REQUIRE WILL INVOLVE THE PROJECT 
RESOURCE DATA BANK. 



*'the subjective assessment of 
over one hundred courses which 
students have taken are compiled 
In two notebooks to assist fu- 
ture students tn program planning* 
A coordination of the pRDB and 
the new evaluation Instrument 
has occurred during the past two 
months to insure continuity of 
services.*" (Cjt E, p.^) 
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Th^ PROB engaged in a series of eval- 
uational activities to assess its users* 
needs. Ani<->ng th'^se were structured interviews, 
quest ionna i res J frequency tabulations of use, 
etc« It has also directed and maintained 
the systematic evaluation of courses. An 
external evaluation of the Data Banks' utility 
for planning student progranns, among its other 
functions, was undertaken by an information 
special ist, 

A major function of the PRPB involved 
trying to facilitate planning of student 
programs. For example the PROB had: 

0 set up and maintained a file on 
individual consortium members* 
It was composed of data on an 
array of variables, such as 
consortium training needs* 
2) set up an ongoing course evaluation 
file containing such data as in- 
structional mode descriptions, 
lecture outlines and course rela- 
tionships to the Taxonomy of Content 
and Ski 1 1 s , etc. 
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3) conducted a needs assessment of 
existing instructional materials 
which yielded a cadre of compli- 
mentary articles and books and ^ 
great deal of information increasing 
the awareness of available materials^ 
^) identified and maintained a file 
on evaluation or evaluation related 
university based courses* Df 
those, approximately thirty were 
evaluated with regard to course 
attributes. Other opportunities 
for training were also sought, 
identified, and evaluated. These 
wet'e kept In folders for potential 
use In planning student programs* 
^ letter from HEW, July, 1971, to the 
Project Director synthesized the previous 
USOE site visit and noted the utility of the 
PRDB to the Model Training Program* John 
Egermeier reported that the reviewers felt 
the Data Bank concept had 9reat utility, 
especially In a consortia. 

However, a later site visit proved to 
be less favorable to the PRDB's function. 
John Egermeier, commenting on the PRDB's relevance 



F 

p. 23 



F 

p. 28 



t6 
F 

"It would be good, destreable, 
and Imperative that these dif- 
ferent views be reconciled so 
that the PRDB facility In fact 
serves the needs of the training 
program.'* {D$, F, p. 2) 



p*2 
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to the program during the October, 1971* exit 

interview, indicated that there were discrepant 

views of the PRDB's function within the program. 

He said the site team noted two aspects of the 

PRD8: I) that it had a library function, and 

2) that it had an information function to serve 

the faculty, the students and the administrators 

of the Training Project, It was recoiwnended 

that the MTP split off as much of the library 

function as possible and i;-.e the O.S.U. 

library staff and facility for that purpose* 

(G) THE MATERIALS AKD SYSTEMS THIS WILL 
REQUIRE WILL IKVOLVE PROCEDURES AND 
MATERIALS FOR PLACEMEKT OF STUDEKTS, 
'TOTH IN IKTERKSHIPAND REGULAR JOB 
SITUATIONS. 

A synthesis of a l97l USOE site visit 
recocwnended the internship and work experience 
opportunities were a strong component of the 
MTP< One suggestion for improvement, however, 
entailed imposing a bit more structure on the 
procedures for i ns true t iona I purposes . 



H 

G 

p.2 

"Your internship and work ex- 
perience opportunities are a 
strong component of your program. 
It Is possible that these situa- 
tions may be improved if they 
are rr^ore structured (engineered 
internship idea) for instructional 
purposes," (D^, G, p-S) 



A number of student and adjunct professor C 

G 

assessments were undertaken in order to more 
closely tailor the students' career goals to 
agency needs for internship and/or full-time 
placement. For example, a three page unsyn- 
thesized evaluation of MTP student needs was 

24 
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done regarding internships and was reported 
in tabular form. Students were largely 
queried regarding their 1) internship plans, 
2) internship awareness^ and 3) ideas soli^ 
cited on how information regarding internships 
could best be publicized* 

During the December, 1372, consortiun C25 

G 

meeting^ adjunct professors were requested to p* 
list the names of agencies similar to their 
own relative to goals and products* A wide 
array of organizations^ centers, etc., were 
generated which gave the project additional 
leads for pof-entlally placing students in 
internships or full-time post graduate positions. 

An assessment by the Consortium Liaison 
Director for the PRDB suggested that the PRDB 
adopt a set of objectives, one of which 
corresponded to student internships* 

The Student Survey attempted to acquire 
some data regarding the students' perceptions a 
of internships and placement* These data 
revealed a lack of student knowledge about 
the consortium. 

25 
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**Based on infonnatlon collected 
by Diane Relnhard on the PRDB 
questionnaire and conversations 
with agency personnel, the fol* 
lowing objectives are suggested*' 
. . *one of these was "to provide 
agencies with (with students ap- 
proval) information on students 
seeking internships*** (C^, G, p*2) 




*The student perception of the 
consortium coupled with the open 
reluctance of students to go out 
into internships expressed in the 
Purdy, Lash, HcFarland Internship 
paper, indicates a basic contra* 
diction with the basic project 
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In an attempt to increase the accuracy 
of planning for student placement, students 
were asked about their career projections* 
The data were synthesized In regard to the 
frequency o? MTP students that saw themselves 
becoming evaluators^ Out of eighteen students 
interviewed, five Indicated that they expected 
to become evaluators, three Indicated 'Vnaybe", 
and ten said ''no"* Subsequent to making this 
estimate, the Process Evaluator cross-checked 
the estimate with responses on a previous 
product evaluation undertaken by a different 
evaluator* These data revealed that out of 
nineteen students^ eight projected that they 
would become evaluators, six said '^aybe'\ and 
ffve said *'no'\ 



objective to create field*based 
training for evaluators to be 
trained tn the rual world of 
evaluators." {Cig* C, p.TS) 

Some student response^ regarding 
the consortium, Included in the 
report were: 

1) "I don't know very much about 
It." 

2) 'The consortium agencies are 
not doing their share**' 

3) 'The Internships are not 
meeting our needs*" 

k) "lt*5 a great idea, but It*s 

not working." 
{C|8» C, p. 15) 

G 

p.l'* 

"tabulations suggest that the 
current student selection proce- 
dures are partially effective, 
and that an Improved procedure 
would probably serve a tritlcat 
project objective*" (Cjg» G. p-l**) 
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MTP staff present during the December^ 
)972» consortium meeting, indicated a need 
for MTP students to flow into agencies to make 
room for agency people to come tnto the 
university. However, the evaluation of field 
agency adjunct professors indicated that the 
MTP is faced with too many students on board 
to place on an individualized basts in 1972* 

Very little, If any, formal evaluation 
occurred in regard to full-'time student place- 
ment. This was undoubtedly related to the 
fact that the greatest proportion of students 
were not facing that decision point during 
the program's duration* 

(H) THE MATERIALS AND SYSTEMS THIS WILL 
REQUIRE WILL INVOLVE COGNITIVE AND 
AFFECTIVE BEHAVIORS THAT STUDENTS 
AT EACH LEVEL OF THE TRAINING 
PROJECT WILL BE ABLE TO EXHIBIT 
FOLLOWING COMPLETION OF TRAINING* 

Because the MTP never institutionalized 

an evaluation design, specifications by which 

it assessed it's acconip I ishments were largely 

inadequate* This unfortunately was true of 

student assessment. Of necessity, student 

evaluation proceeded in an ad hoc manner 

and was fragmentary* The multiple criteria, 

which was in the process of being operation- 

alized in the SAES instrument, wu not ready 
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for actual use. Consequently, formal cognitive 
student assessment was never executed. There 
were, however, a series of affective measures 
used throughout thi; project's duration to tap 
student attitudes* These were administered 
upon entry, during interim periods, and upon 
exiting the program* 

, Several student assessments were under- C]y 



taken solely to acquire descriptive profile 
data* One such undertaking reported tn tabular 
form, was compiled early in the program and 
contained information on ^he following areas: 
I) degree sought, 2) age, 3) sex, k) membership 
rn minority group, 5) grade point average, 
6) previous work experience, 7) honors, and 
8) publications. 

Early in the HTP, during the December, Ejq 



ineffective for both planning and diagnosing 

student development* Within a period of 

eight weeks, a decision was made to adopt 

the CIRCE Attitude Scale to assess the students* 

attitudes toward evaluation and a complimentary 

semantic differential was developed at this time. 



staff, assessing the system for obtaining 



cognitive and affective data on students. 



indicated that the present procedure was 



1971, consortium meeting, 0*S*U* centered 
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The CIRCE Attitude Scale and the semantic 
differential were two structured instruments 
which were adopted and applied for the purpose 
of assessing MTP student attitudes toward 
1) evaluation in general, 2) the improvement 
of education from evaluation efforts, and 
3) the student *s self perception of his eval* 
uational capabilities. These were administered 
upon entry and scheduled for reappi ication upon 
the students' exit. The sudden curtailment of 
program development naturally precluded a 
cofiiplete assessment of exiting students* 
Pre*testing results were interesting since 
students generally had had only limited ex- 
posure to evaluation, and yet they evidenced 
a surprising degree of convergence regarding 
their attitudes toward evaluation. Post- 
testing revealed a discernable shift in atti* 
tudes among exiting students. 

Responses to two semantic differential 
scales administered with the CIRCE Scale 
indicated that students felt that the state 
of the art (ie*, evaluation) was still Inade- 
quate* The wide divergence of response to the 
Instrument precluded a meaningful synthesi*^. 



ho 

H 

pj 

"A noticeable shift In student 
attitudes occurred on 1) the 
judgement scale-*exit Ing stu* 
dents showed greater receptivity 
to judging the worth of a 
pro94am as perceived by the 
program affiliates than they 
did during the pre-test, and 
2) the research scale—exiting 
students were more receptive to 
relying on precise measurement 
and statistical analysis to 
gain general understanding of 
why programs do or do not succeed 
Students also exhibited an in* 
creased receptivity to the ob- 
jective orientation to evaluation 
Downward trends are notable In 
"evaluation as a service orienta- 
tion*" (C20, H, p, ) 



C2O 
H 
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The project additlonaUy provided for 



C20 
H 

'The respondents Indicated that 
a need for assessment of student 
capabtUt{e$ during entry would 
allow each student to adjust 
smoothly to the program 
requirements J* (C2o» H, p. ) 



the collection of Informal open ended Inter- 



views from exiting MTP students concerning 



their attitudes toward the total program* 



The majority of questions focused on student 
perceptions of program adequacy and their 
ImmectFate and future plans* Respondents 
Indicated that their most productive, program 
affiliated, learning experiences were non- 
classroom activities. The majority of negative 
experiences were associated with the student's 
Initial contact with the program* However, 
most respondents Indicated that these dissipated 
within a short time* In the cognitive area, 
a need for diagnosis during entry was expressed. 
Additionally, the majority of respondents 
Indicated a lack of technical competencies 
resulting In a feeling of discomfort In 
anticipation of an evaluation position* 

There was dissatisfaction with the C20 



program Itself* At least half of the students 



tations prior to entry and their actual ex- 



Indlcated a discrepancy between their expec- 



H 
p. 

'They expected a program which 
achieved a higher degree of 
sophistication In development 
than they found." {C20» H, p. ) 



per iences* 



"Student orientation to the MTP 
ts not always tailored to meet 
Individual needs and tolerance/* 



(C20' H, p. ) 
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During the Oecembier, l972, consortium 
meeting, adjunct professors were requested to 
suggest instrumentation and focal behaviors 
for assessing intern performance during their 
enrollment in field based training. Most 
responses were vague, suggesting areas such 
as the students evaluation skills, his "sense" 
of responsibility, human relation skills, etc. 
Some of the more specific suggestions entailed 
a focus on the student's acquisition of cog- 
nitive knowledge, such as knowledge of I) who 
is who in the unit and parent agency, 2) mea- 
surement, 3) reporting, and ^) design. 
Suggestions regarding the means of assessin9 
these skills varied from student and staff 
self-evaluation to interviews, questionnaires, 
and the application of structured field 
observa t ion schedu I es . 
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The Model Training Program was terminated too early, or it was 
implemented too late, to allow a sound evaluation of its implementation 
at this point of phase out* Together with the most enthusiastic 
impressions of outside reviewers such as Egon Cuba, Michael Scrrven, 
Frank Chase and OE site visitors, there ts partial evidence of various 
problems in the process of implenentation* One possibility is to 
perceive this partial evidence as an indicator of some other problems 
that were never uncovered; another possibility is to disregard it in 
the light of the highly significant efforts made by this program 
during tts short life span. 

The following presents some "fractions of evidence", regarding 
the implementation of the Model Training Program, according to the 
Standards defined for this Objective* 



General Objective 3 : TO IMPLEMENT THE GENERAL 
MOOEL FOR TRAINING EOUCATIONAL EVALUATORS, 



(A) A SIZABLE POOL OF POTENTIAL TRAINEES 
FOR THE MODEL TRAINING PROJECT WILL 
BE lOENTIFIED. 



of March 2k, 1972, we find that '^a pool of 



In the "In-House MTP Quarterly Report 



In-House MTP Quarterly Report , 
March 29, |972 

P*7 



85 otential trainees ha^ been identified 



which is for in excess of the number the 



program can accep 



tJ' 
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Mo information is available on the quality 

of the potential trainees or the procedure used 

to identify them* 

(B) QUALIFICATIONS OF APPLICANTS FOR 

PARTICIPATION tN THE MODEL TRAINING 
PROJECT WILL BE ASSESSED TO IDENTIFY 
THOSE STUDENTS WHO ARE MOST LIKELY 
TO PURSUE AN EVALUATION CAREER AND 
PERFORMABLY IN SUCH A ROLE. 

An internal evaluation report of March 2k, 

(972, evaluates this standard as 'V^arttally 

met" due to the lack of appropriate procedures* 

The OE site visitors in October, 1971* were 
satisfied with the quality of the HTP students 
but have questioned some of the recruitment 
criteria: 'There is a long tradition of 
training only educators in education. How- 
ever, there are a )ot of people advocating 
that we can pick up a lot of strength by 
increasing the diversity of our groupJ* 

In the fornrtal letter from OE to the 
Director of the Evaluatonn Center, following 
the site visit is stated: 'Mt appears that 
you are solving many problems associated with 
the recruittnent and selection system, but we 
would like to urge that you develop the system 
to also aid In the recruitment of people outside 
of education and outside of the consortia agencies 



In-'House Quarter_ly Report , 

March 2k, ]372 

C| 

*Mt will be part of future task 
force effort relative to recruit- 
ment and advisement procedures."^ 

D| 

"Our impression was they {the 
students) are alert, capable, 
and we don't have any concern 
about their being able to perform 
in the rotes for which you are 
training people," (oE site visit, 
October 26-27, (971) 



06 
p,l 

A letter to Dr. Daniel L* 
Stufflebeam, from Drs, John C 
Egermeier and Susan S* Klein, 
November I7, |97K 
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(C) THIS WILL INVOLVE SELECTING OUT- 
STANDING APPLICANTS FOR TRAINING 
WHO ASPIRE TO BECOME GENERALIST 
EVALUATORS, DIRECTORS OF EVALUATION 
OFFICES PROFESSORS OF EVALUATION, 
AND EVALUATION-ORIENTED EDUCATIONAL 
LEADERS 

The In-House MTP Ouarterly Report states 
that the *'eva I uat ion roles'' are unclear and 
it IS still early decide what actual posi- 
tions the students wi H fulfill after they 
graduate . 

This report presents also the following 
data taken from a student questionnaire that 
asked about MTP students* plans for the future; 
evaluation career - 8 students 
non-eva luat ion career * 6 students 
undecided - 5 students 

(D) EACH STUDENT WILL BE HELPED TO DIAGNOSE 
HIS TRAINING NEEDS, TO PROJECT A RELE- 
VANT PROGRAM OF UNIVERSITY AND FIELD 
TRAIHIHG EXPERIENCES, AND TO INTERACT 
SYSTEMATICALLY WITH A COMMITTEE OF 
FIELD AND UNIVERSITY BASED PROFESSORS. 

In the In-House MTP Quarterly Report of 

March 2^, l972, this standard Is reported as 

**not met** a I though efforts are made to develop 

procedures that seem likely to succeed. 

In a student survey reported at March 28, 
1972, it was found that "about one half of the 
interviewees (n=l6) felt there was little to 
no formal advisement procedure." 

34 
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P-8 

'The efforts of Bob Lange, as 
interim students advisor, and cf 
Mary Anne Bunda, Darrell Root, and 
Bob Lange in onducting the MTP 
seminar has resulted in positive 
student feedback about the problem 
solving efforts.*' 

^18, 
P. ^ 

**One of the reoccuring statements 
was that new Center staff do not 
know enough about university 
offerings, , .advisors outside the 
MTP cannot give clear direction 
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relative to evaluation goals or 
the specific need of HTP students'* 

Eva Baker observed that "the number and Dg 

p. 67 

function of formal coursework has not been decided Or. Eva Baker, HTP Outside Review 

Panel, visit of Juiy 29-30, I97i 

upon", whether offered by the university or by 
off campus adjunct professors. She suggests 
that "procedures for consortium-wide consider* 
ation of coursework should be given accelerated 
development." 

Chase suggested that "the differentiation of 

p. 9 

roles and the ; (location of responsibilities Report by Francis S. Chase on the 

site visit to the HTP of the 
among the regular university faculty. Evaluation Evaluation Center of OSU* 

Center staff, and the adjunct professors requires 

further attention." 

On the OE site visit of October, 1971, John 0] 

p.2 

Hopkins made the fol Iw-Ing observation: OE Site Visit, October 26-27, 1971 

^\ . .as I talked with students t was not able 

to discern that you had as yet put into effect 

the individual diagnosis, individual counseling, 

and individual programing of students that 

promises to be one of the really important 

characteristics of this program." 

John Peper, on the OE site visit of D)q 

p. 3 

October, 1972, made the following statement; John B. Peper, Chairman's Report 

of the site visit to OSU HTP, 
"All of the review team observed that the October l3-l^, 1972. 

student training model developed by Bargar was 

excellent. We were dismayed to find that students 
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were not being given an opportunity to exper- 
ience the fu)\ potential of the model because 
of overemphasis on project assignments,** 

(E) STUDENTS WILL PARTICIPATE IN 
SPECIALLY DEVISED SEMINARS AND 
PARTICIPATE MEANINGFULLY IN 
EVALUATION WORK EXPERIENCES. 

The In-House MTP Quarterly Report of March 
2^, 1972, states that this standard was partially 
met: *^scme seminars available and more to come; 
the Work experience is seen as least desirable 
by most students because It is mostly develop- 
mental instead of evaluative** {Cl, p. 9) 

A similar opinion wcs expressed by Eva 
Baker* She criticized the involvement of 
trainees in material development within MTP. 
'H'he expected dependence upon trainees to 
perform tasks required for the development of 
the Model Training Program itself, e,g., for 
material development, seems unwise.*' Although 
her main concern is the quality of materials 
that will result from the involvement of 
students in their development, she also ques- 
tions the merit of such experience for students 
as part of their training program* 

A completely different opinion is expressed 
by Chase in his report on the s te visit to 
MTP* He points out the experience provided at 



Cl 
P.9 

In the student survey {reported 
March 28, l972) **nine out of 
thirteen students felt that their 
Center job had Il^'tle or no 
bearing on their long-term eval- 
uation goals*** (C23, p*7) 



D8 

Dr, Eva Baker, MTP Outside Review 
Panel, visit of July 29, 30, 1971* 



D9 
p*I 

Report by Francis S* Chase on the 
site visit to the Model Training 
Program in Evaluation of the 
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the Evdtu3tion Center as one of the notable 
strengths of the training program* "Perhaps 
the most significant aspect of this setting is 
that it will enable the trainees to become active 
members of a group that is deeply committed to 
improvement of the concepts and technotogies 
of evaluation in education*" 

The fact that students are involved in 
developmental experiences is perceived by Chase 
as another source of strength in the HTP: 
'Trainees themselves wi 1 1 be actively involved 
in creating and improving a program of training 
instead of being treated as recipients of ideas 
worked out by others*" 

This idea fs stressed also by the OE 
site visitors of October, 1971 ^ 'V/e endorse 
the concept that you have developed for this 
program of using the students in the develop- 
mental capacity of refining, developing, and 
Improving their program. 

But the OE site visit of October 1972 came 
to a different conclusion* in the report of 
the chairman of this team we read; "(t is 
further recommended that less program develop- 
ment responsibility be given to students, * * 
priority should be given in those tasks to 
developing and testing the content modelV" 
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Evaluation Center of OSU. 



C9 
p. 2 

Report by Francis S. Chase on the 
site visit to the Model Training 
Program in Evaluation of the 
Evaluation Center of OSU* 



Dl 
pj 

'V/e would join Egon Guba tn urging 
you, or perhaps it was Frank Chase 
to continue to use the students In 
this capacity even after you h^ve 
a first cut program* 



DlO 

(Peper) 
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Sore explanation of the discrepancy between Oy 

p*6 

the opinions of outside reviewers and HTP Michael Scriven^ hTP Outside 

Review Panel, Report of 
students may be suggested by Scriven, inspite September 7^ 1971. 

of his general impression of the MTP as an 

''extraordinary well-thought out and well 

implemented organization plan**, he found that 

**there appears to be inadequate success in 

explaining to the student the significance 

of the work he undertakes as part of his 

training.** 

(F) STUDENTS WILL EXHIBIT DESIRED 
TERMINAL BEHAVIORS AND ATTITUDES 
FOLLOWING COMPLETION OF THEIR 
TRAINING PROGRAM, AS SPECIFIED 
IN TH£ OVERALL TRAINING MODEL. 

This standard has not been assessed. 

(G) STUDENTS WILL BE PLACED IN 
SATISFYING PROFESSIONAL POSITIONS 
FOLLOWING COMPLETION OF THEIR 
TRAINING PROGRAMS. 

This standard has not been assessed yet. 
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General Ofaiective k : TO DEVELOP A CONSORTIUM 
OF FIELD AND UNIVERSITY AGENCIES THAT WILL 
IMPLEMErfT THE TRAINING HODEL IDErfTIFlEO ||J 
OBJEaiVE K 

(A) THE CONSORTIUM WILL INCLUDE AGENCIES 
THAT HAVE TAKEN LEADERSHIP IN EDUCA- 
TIONAL EVALUATION AND THAT REPRESEKT 
SCHOOLS, STATE EDUCATION DEPARTMENTS, 
RESEARCH aND DEVELOPMENT INSTITUTIONS, 
NATIONAL EDUCATIONAL AGENCIES, AND 
UNIVERSITY AGENCIES. 

The acceptance of an agency into the 
consortium was contingent upon meeting a list 
of criteria developed by the Model Training 
Project. Agencies were contacted to see if 
they were interested tn becoming a member of 
the consortium and in cases were there was an 
indication of interest a visit to the agency 
was carried out to determine the agencies 
evaluation capability and commitment. 

Outside reviewers commenting on the 
strength of the consortium generally gave 
high marks to the consortium membership. 
Several recommendations were made to improve 
the representation of various educational 
agencies within the consortiun. It was the 
concern of several reviewers that the consortium 
Include one or two state departments. Although 
several were reviewed, only the State Department 
of Michigan was finally accepted. With the 
acceptance of this one state department all 
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p. 22 

The reviewers noted that you seem 
to be developing a strong, viable 
consortia and urge that you con- 
tinue your efforts in this dlrectlo 
by using your newsletter, liaison 
person, well defined contracts. 
Adjunct Professors and other 
resources or mechanisms which 
may prove beneficial in strength- 
ening the consortia. 

p. 3 

This observer shares that feeling 
and believes that further informa- 
tion sKould be sought from the 
Office of Education, the Com- 
mission of the States, and other 
sources in regard to the corniit- 
ment and capability of the several 
state departments for planning 
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of the agencies specified In the above standard 
were included in the consortium^ 

In enumerating the responsibilities of 
a consortium liaison officer, Guba suggested 
that such an individual could assist in the 
recruitment of consortium menters. However, 
the major responsibilities of the appointed 
officer was to help provide communi cavion 
linkages within the consortium^ Evaluations of 
those responsibilities are included under a 
different standards 

The site visits revealed that there are 
some agencies within the consortium Incapable of 
fulfilling the criteria for Consortium maimer* 
ship* For example, the Council of the Great 
City Schools had no evaluation unit and 
apparently the staff did not support the basic 
concept of evaluation* 

It was the judgement of several outside 
evaluators that the consortium represented the 
strongest part of the overall program* In 
both the I97I and 1972 Federal Site Visits the 
development of the consortium for the Model 
Training Project received strong support. It 
was the concensus of the reviewers that the 
institutions In the consortium represented a 

40 



and evaluation* Out of this 
search five or six likely 
prospects might be identtfed and 
from these prospects two selected 
as added mechbers of the Consortium* 

p. 8 



The strongest single element of the 
OSU *<odet Training Project Is the 
consortium development. This 
developmental effort attests 
dramatically not only to the 
feasibility of consortium notion, 
but also to the exemplary credit 
due Or^ Stufflebedm and his 
management staff* 
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diversifted membership and that they had 

become an interna) part of the operation 

providing useful input as to the direction 

of the project* 

(e) THE CONSORTIUM WILL DEVELOP A 
. NATIONAL FACULTY OF HIGHLY 

QUALIFIED EVALUATION LEADERS WHO 
CAN OFFER BOTH UNIVERSITY AND 
FIELD BASED TRAINING AND WHO ARE 
THOROUGHLY ACQUAINTED WITH AND 
INVOLVED IN THE DEVELOPMENT OF 
THE MODEL TRAINING PROJECT. 

The general reaction to the national 
faculty of adjunct professors was very 
favorable* The major concern of reviewers 
revolved around the relationship between the 
central project staff and the consortium 
members* Guba considered this to be one of 
two major factors which could make the MTP 
an outstanding success^ However, he felt that 
the responsibility for insuring a proper 
relationship fell mainly on the staff, {It 
was partly this reasoning which led him to 
suggest a Consortium tralson Officer). A 
proper relationship required that the adjunct 
professors perceive their roles in such a way 
as to involve the members and full facilities 
of their organizations in the training program. 



^7 

p: 13 

The Adjunct Professor Idea Is an 
exciting new try at making work- 
study part of the academy* 

p. 8 

h 

FTg. 1 
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The desire to actively involve adjunct 
professors in the training program ]ed to a 
series of activttTes in which these representa- 
tives of consortium organizations were brought 
to the Center to work on specific development 
tasks. In addition, the adjunct professors 
were involved in work efforts during ConsortTIum 
Decision making meetings. The last of these 
meetings {July, 1972) proved in the opinion 
of the adjunct professors to be generally 
productive* Agenda changes leading to a more 
productive involvement in this latter meeting 
resulted from expressed dissatisfaction wltli 
the previous agenda* 

Although an attempt was made to work 
closely with the adjunct professors in ofder 
to build a sense of Identity wi th and a 
commitment iS the program there was a lack of 
systematic effort In accomplishing this purpose* 

(C) THE CONSORTIUM WILL BE BOUND TOGETHER 
BY APPROPRIATE LEGAL AGREEMEm-S. 

This standard was met in that the 

consortium was bound together legally by 

cont ractual agreements. 

(d) the consortium will design and 
implemem" a feedback network among 
consortium members that will ma|m"ain 
good communication within the consortium 
and among the members of the national 
evaluation faculty* 
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The need to design a communi cat tons 
network was delineated early in the program and 
several suggestions were offered to meet it. 
The suggestion was made that a consortium news- 
letter be established as an immediate linkage 
between members. In addition, an Occassional 
Paper Series concerned with relevant conceptual 
Issues was proposed as a means of keeping 
members informed and also as a vehicle for 
genuine contributions to scholarship. The 
major recommendation proposed to strengthen 
communication between agencies was the appoint* 
ment of a Consortium Development Staff Officer 
(Consortium Liaison Officer). This officer 
would have the following responsibilities: 

1. General liaison with consortium members. 

2. Building a sense of commitment and 
identity within the adjunct professor 
group. 

3. Providing information to the central 
staff and the members about each of 
the agencies. 

The above recommendation was given wide 

spread supported by the central staff, adjunct 

professors and the Office of Education. A 

self evaluation by the Consortium Liaison Director 

provided the following assessment: 

U Frequent and intense contact with 
adjunct professors was not possible 
due to budgetary and time limitations. 

2. Involvement of adjunct professors in 
HTP activities is limited* 
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Intervisltation, consultation, 
and communication among Con^* 
sortium members should involve 
not just one but many individuals 
from each member organization if 
the full potential of the Con- 
sortium is to be realized. 

P- 5 
P- 9 

It is my recommendation that 
fult-'time Consortium develop- 
ment staff officer be appointed 
at once to undertake this 
vital mission. I regard this 
recommendation as the major 
recommendation that I will make 
anywhere in this report. 



p! 6 

We agree it was a good move as, 
Mr. Guba suggested, to appoint 
the Director of Consortium. I 
won't use the common title, but 
this is Diane's position. And 
we think that should serve to 
keep the consortium strong, help 
it to grow, and to maintain the 
influence which it apparently 
has now. 

^10, 
p. k 



3* The gathering of information from 
consortium agencies for the central 
staff was mtnor* The major emphasis 
tn this activity was in the identification 
of internship positions. Site visit 
reports and a portfolio of information 
on each agency was distributed to the 
various agencies. 

Although the Liaison Officer was limited 
in her activities by funding and time con* 
straints the adjunct professors considered 
her activities hetpfut to them as members of 
the consortium. Overall they expressed 
satisfaction for the manner in which the 
position was handled. 



The consortium newsletter was established 
and a survey of seven adjunct professors 
indicated that six read the newsletter and 
found jt useful . 



C| 

p. 13 



The quarterly consortium meetings 
provided a direct opportunity to communicate 
information and assess the adequacy of that 
cofwnunication. The December, 1971 meeting 
resulted in dissatisfaction on the part of the 
adjunct professors with the agenda. Feedback 
for the adjunct professors resulted in several 
constructive suggestions being offered and 
adopted. Particularly it was decided to involve 



pI°15H6 

Adjunct Professors should be 
consulted about adenda items^ 
Possible Darrell Root should 
call each meoiber I week prior 
to each meeting to confirm 
and reach concensus on agenda 
i tems. 
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the adjunct professors to a greater extent 
tn the planning of agendas and to minimize 
activities which could be handled outside of 
tne consortium meeting. 



Student perceptions concerning the con- 
sortium differed with those of the fulltime 
staff. An evaluation of student attitudes 
revealed a tack of Information concerning the 
consortium* It was suggested that the 
Consortium Director also address student 
information needs as welt as Center and 
inter-^gency needs. 

(E) THE CONSORTIUM WtU SERVE AS A 
MECHANiSM FOR THE RECRUITMENT OF 
ABLE STUDEflTS im"0 THE FIELD OF 
EVALUATION, 

A total of five students were recruited 
througn the consortium agencies. Mechanisms 
were developed to help identify students in 
consortium agencies and the Consortium Liaison 
Officer spent a great deal of time attending 
to student matters. 

The consortium agencies were not able to 
supply enough students which necessitated the 
recruitment of students from outside the field 
of education. This, however, proved to be 
a desireable activity and one which was supported 
by USOE. 45 



^|8._ 
P* 15 

The following are a series of 
student statements that reveal 
a lack of Information about the 
Consortium: 

K.*M don't know very much about 
it.»» 

2. "The Consortium agencies are 

not doing their share*** 
3- *The internships are not 

meeting our needs*** 

** It* s not useful*'* 
5* *Mt*s a great Idea but it*s 

not working*** 



^10^ 
p. 6 

Close examination of Consortium 
Liaison DIrector*s Site Visit 
Reports seem to reinforce my 
perception that the majority 
of my time ts spent In student ' 
matters at the agencies* 



p* I 

It appears that you are solving 
may problems associated with the 
recruitment and selection system, 
but we would like to urge that 
you develop the system to also 
aid In the recruitment of people 
outside of education and outside 
of the consortia agencies. 



(F) THE CONSORTIUM WILL SERVE AS 

A FIRST-LINE NETWORK OF AGENCIES 
FOR PLACING TRAINED EVALUATORS. 

An evaluation of potential agency staff 

needs indicated that approximately 3^ positions 

would be available between l972 and 197^* 

The majority of these positions were to be 

aval lable in 1973. 

(G) THE CONSORTIUM WILL, PROVIDE SITES 
FOR FIELD TESTING OF THE GENERAL 
TRAINING MODEL AND SPECIFIC TRAINING 
MATERIALS. 

No evaluation is available on field 
testing carried out in the consortium. How- 
ever, the following materials were field 
tested in one or more agencies* 

K the recruitment brochure 

2. the Student Handbook 

3* the "Institutionalization of Evaluation** 
seminar package* 

(H) THE CONSORTIUM WILL PROVIDE A NETWORK 
OF AGENCIES THAT ACCEPT, SUPPORT, AND 
IMPLEMENT EFFECTIVE EVALUATION PROCE- 
DURES* 

The degree of effective evaluation 
implementation has been assessed minimally by 
the site visits to consortium agencies* The 
synthesis of these assesments are more properly 
presented under Objective 5* 

(I) THE CONSORTIUM WILL DEVELOP AND 
IMPLEMENT LONG-RANGE PLANS FOR COLLABOR- 
ATIVE EFFORTS OF THE CONSORTIUM AGENCIES 
IN THE IMPROVEMENT OF EVALUATION THEORY, 
PRACTICE, AND TRAINING* 

No evaluation is available* AH 



General Objective 5 : TO ASSIST AGENCIES 
OF THE CONSORTIUM TO INSTITUTIONALIZE 
EVALUATION THROUGH PLACEMENT OF MODEL 
TRAINING PROJECT GRADUATES AND THROUGH 
IN-SERVICE TRAINING ACTIVITIES IN THE 
CONSORTIUM AGENCIES* 

(A) SUCCESSFUL INSTITUTIONALIZATION 

EFFORTS WILL BE CHARACTERIZED BY THE 
ABILITY OF THE MODEL TRAINING PROJEa 
TO TRAIN ADDITIONAL ON-SITE PERSONNEL 
VIA IN-SERVICE TRAINING IN EACH OF THE 
CONSORTIUM AGENCIES* 

Although the training of on-site 

personnel was gaining a great deal of momentum 

In the tatter stages of the project there was 

little attempt to evaluate the effectiveness of 

this training. Many seminars were being offered 

by adjunct professors and subordinates either 

on their own Initiative or in response to 

materials developed by the staff at 0*S*U* 

Special inservice training was provided at 

Shanty Creek in the inst i tutionat ization of 

evaluation, tn addition, 70 central office 

administration personnel and building principles 

received special in-service training at Saginaw, 

Michigan* 

Evaluation accomplished under this stand- 
ard was mostly in the from of a needs assess- 
ment* Several evaluations discussed the 
necessity of inservice training and specified 
the direction in which it should go* The 
need to provide inservice training which 
focused on actual agency problems was 
frequently underscored* 



^7 

pl 57 
^9 

pi 10 

How effective are the provisions 

for continuing staff develop* 

ment of the organizations In 

the Consortium? tt seems to 

me that one of the most Important 

returns on the Investment 

which Consortium members make 

in the training program should 

be the upgrading of the competence 

of their own staffs in evaluation* 



I would venture to submit*** 
that all adjunct professors at 
alt agencies would prefer short 
inservice type training (three 
days) and would prefer limits 
of seminars to actual dis- 
cussions of problems within 
internships* 
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(8) SUCCESSFUL INSTITUTIONALIZATION 
^ EFFORTS WILL BE CHAFIACTERIZED BY THE 
ABILITY OF THE MODEL TFlAlNING PROJECT 
TO ASSIST CONSORTIUM AGENCIES TO 
DIAGNOSE AND RESPOND TO THEIR 
EVALUATION NEEDS AND PROBLEMS. 

Initial response to this standard was a 
needs assessment which pinpointed probleni 
areas in the Institutionalization of evaluation 
In addition each agency was screened prior to 
admltance to determine the degree to which 
they had established a viable evaluation 
capabi I ity* 

During the Shanty Creek Meeting each 
adjunct professor prepared a plan for the 
institutionalization of evaluation in his 
agency and assessed his plan against a paper 
by Guba and Stufflebeam concerning strategies 
for installing evaluation systems* A synthesis 
of common concerns and strategies was prepared 
and distributed to all adjunct professors to 
assist them in achieving the goal. 

A Survey of adjunct professors designed 
to determine whether help had been received 
from the MTP In diagnosing evaluation needs 
and problems produced a variety of responses* 
Most respondants cited the site visit as a 
source of help and tended to view the visit 
as a catalyst for constructive action* One 

48 



agency, in close proximity to the Evaluation 
Center indicated that its staff received 
specific informal input in the form of 
consultations. The Guba, Stufflebeam paper 
on "Strategies for Institutionalizing the CIPP 
Hoder* was also cited as a destreable input* 
One agency specifically indicated that no 
input had occurred. 

Site visits were carried out by the 
HTP at the following agencies. 

1* Ssgtnaw Public School System 

2. The Council of Great City Schools 

3. Institute for Educational Develop- 
ment 

^. Cincinnati PubPc School System 

The site visits revealed that the majority 
of agencies had not clarified adequately their 
organizational goals anJ the criteria for 
achievement of those goals. In addition, poor 
internal communications was a common concern 
and problem within the organizations and fre- 
quently contributed to the lack of rlarity. 
A lack of formal policies for handling routine 
procedures was also cited as a frequent problem 
within the agencies. Dependence upon "soft 
money" presented problems which could not easily 




'Assessment by the task force 
team members of goals and the 
priority of goals as related 
to projects the division Is 
engaged in Indicates a vagueness 
in documentation* Further 
investigation of PR & 0 staff 
opera t ions indicates a lack 
of goal direction as a standard 
in guiding project activity." 



"Another weakness noted was 
that fED had not been able to 
secure "hard money" to provide 
a core of support to facilitate 
Systematic long-range planning 
and development. 
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oe solved. Lack of cooMnitmcnt on the part 
of parent organizations made dependence on 
*'soft moneys* a neco'jSity. 

Generally, the strengths of the different 
agencies were found in the commitment and 
energy of the staff. Strong administrattons 
and capable personnel provided a positive 
indication of futvire growth and stability. 
In two agencies, the efforts to establi^n 
strong communication with other departments 
in the organization was commended. 

Specific recommendations to the agencies 
visited were provided on aH major concerns. 
In addition to the recommendations which 
resulted from the problems listed above, 
the site reviev^ers also concerned themselves 
with the fol laving items: 

1) information storage and retrieval 
systems were exa^Tiined and recom* 
mendattons for imp*^ovement included: 

a) making the baseline data more 
accessible to project personnel, 

b) clarifying purposes for infor- 
mation syste-.;s, 

c) improving the overall technical 
capabiltty of available systems. 



^7, 
P,2 

"In the same sense, the task 
force was Impressed with the 
quality of the staff and the 
personnel tn leadership positions 
within the division. The task 
force needs also to mention the 
high regard in which most per- 
sonnel, with whom we talked to 
outside the division, hold for 
the work and leadership of the 
division. . .** 



p. 6 

*There should be some attempt 
to move to a storage system 
that does not require professional 
intervention, and e greater use 
of the computer capabilities." 

h 

"It was folt that the staff of 
the Evaluation Unit was not 
taking full advantage of the 
Model Training Project or that 
the Adjunct Professor on site 
was not providing adequate 
leadership. There was no 
evidence of in"servlce training. 

P 8 

'*Poor Inter-offlce communication 
was obvious. A definite lack of 
protocol of formal channels of 
communication was noted. No 
regularly scheduled feedback or 
sharing of information was 
evident among the staff members." 
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2} Staff training Inadequacies were 
identified in three of the four 
agencies* In general, it was felt 
that agencies were not taking ad- 
vantage of training possibilities 
offered by the MTP. 
3) Communication seemed to be a problem 
shared by most agencies* Two agencies 
were particularly cited for poor 
inter'^off ice cocnmunication. in 
the majority of cases, regular scheduled 
information sharing sessions were 
either absent or inadequate* Recom- 
mendations included a variety of 
formal and informal mechanisms for 
maintaining better communications at 
all levels. 
The Cincinnati Site Visitation Team 
provided the most comprehensive analysis and 
recomnendations for improvement* Critiques 
of their ^'Local School Program Development and 
Evaluation Hodel" and ^'School Information 
System" provided evaluative information on 
two programs which are integral to the 
agencies future viability. In addition, specific 
recommendations were oiado for administrative 



and staff reorganization in order to increase 
work efficiency and goal achievement. 

Assessment of site visit effectiveness C| 

PJ7 

indicated that they were not serving their 
function relative to Objective 6, Institution- 
alization* Crivical problems In instrumentation, 
methodology, and commitment minimized the 
effectiveness of the visits* Specific recom- 
mendations to improve instrumentation and a 
clearer definition of the responsibility of 
participating observers was offered in order 
to insure effectiveness in future site visits. 
However, funding restrictions and the unwitH 
ingness of some agencies to be visited ended 
work in this area. 

(C) SUCCESSFUL iNSTITUTIONALIZATION 

EFFORTS WILL BE CHARACTERIZED BY ^ 
THE ABILITY OF THE MODEL TRAINING 
i>ROJECT TO ASSESS THE HOVEKSNT 
OF EACH CONSORTIUM AGENCY TO WORK 
TOWARD INSTITUTIONALIZATION OF 
EVALUATION. 

The site visits provided the only formal 

ooportunity to assess the degree to which 

evaluation had been institutionalized* Since 

only one site visit v;as accomplished for four 

of the agencies the movement of an agency to'^/ard 

institutionalization was not recorded* In 

addition, the contact with agencies through 
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either the Consortium Liaison Director or the 
Quarterly Consortium meetings was not used to 
measure such a movcnient. 

Comprehension of the institutionalization 
problems to be dealt w!th b^ the agencies moved 
from "undecided" to *'agree" on a 1il<ert scale 
instrument administered at the Shanty Creel< 
and Columbus (71) consortium meetings. 
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